Nonconvex proximal splitting with computational errors∗

Throughout this chapter, ‖·‖ denotes the standard Euclidean norm. Problem (1) generalizes the more thoroughly studied class of composite convex optimization problems [30], a class that has witnessed huge interest in machine learning, signal processing, statistics, and other related areas. We refer the interested reader to [2, 3, 21, 37] for several convex examples and recent references. A thread common to existing algorithms for solving composite problems is the remarkably fruitful idea of proximal-splitting [9]. Here, nonsmoothness is handled via proximity operators [29], which allows one to treat the nonsmooth objective f + r essentially as a smooth one. But leveraging proximal-splitting methods is considerably harder for nonconvex problems, especially without compromising scalability. Numerous important problems have appealing nonconvex formulations: matrix factorization [25, 27], blind deconvolution [24], dictionary learning and sparse reconstruction [23, 27], and neural networks [4, 19, 28], to name a few. Regularized optimization within these problems requires handling nonconvex composite objectives, which motivates the material of this chapter. The focus of this chapter is on a new proximal splitting framework called: Nonconvex Inexact Proximal Splitting, hereafter Nips. The Nips framework is inexact because it allows for computational errors, a feature that helps it scale to large-data problems. In contrast to typical incremental methods [5] and to most stochastic gradient methods [16, 18] that assume vanishing errors, Nips allows the computational errors to be nonvanishing. Nips inherits this capability from the remarkable framework of Solodov [33]. But Nips not only builds on [33], it strictly generalizes it: Unlike [33], Nips allows r 6= 0 in (1). To our knowledge, Nips is the first nonconvex proximal splitting method that has both batch and incremental incarnations; this claim remains true, even if we were to exclude the nonvanishing error capability.1 We mention some more related work below. Among batch nonconvex splitting methods an early paper is [14]. Another batch method can be found in the pioneering paper on composite minimization by Nesterov [30], who solves (1) via a splitting-like algorithm. Both [14] and [30] rely on monotonic descent (using line-search or otherwise) to ensure convergence. Very recently, [1] introduced a powerful class of “descent-methods” based on Kurdyka-Łojasiewicz theory. In general, the insistence on descent, while theoretically convenient,